NB: This demonstration analysis was performed on unpublished development data. This experiment was designed to investigate the effect of a number of protocol changes (which were subsequently deemed to underperform) and explore the use of total inputs as a quantitative baseline for discriminating between RBPs more closely associated with non-adenylated RNA than adenylated RNA.
Aims:
A. To test the following modifications to the RNA-RBP Purification protocol
i) Dropping of NaOAc
NaOAc is typically a precipitative agent. It shouldn't be necessary for OdT capture and, if the global capture truly occurs on the basis of salt bridge formation then it should not be necessary for the silane protocol either. Counter-evidence to the salt-bridging hypothesis includes the successful use of salt-free washes prior to elution from silane, and the role of heating in aiding elution.
ii) Dropping of the high pH spike-in (previously delivered as pH10.8 NaOAc)
Use of a pH10.8 NaOAc spike-in was initially used to increase the pH of acidic cocktails to pH7.8. Extensive investigation of RNA degradation has ruled out all components except this one- thus because RNA should be able to withstand pH 7.8, it is likely that the original pH meter measurements of the balanced phenol cocktail were not accurate. In addition, the assition of a DNAse rxn buffer post lysis should be adequate to balance the acidic, but weakly buffered, phenol cocktail stock to above pH7; this has been confirmed by bromocresol green.
iii) Substantively increased neat formamide and tris-buffer wash times for the silane prep
Initial silane experiments have tolerated high degrees of salt-free washing. We are simply revisting this as a convenience aspect of the protocol.
iv) Use of RNAse prior to tryptic digest and increase in both trypsin amount and concentration.
Some past experiments have returned higher than expected missed cleavages. Is it due to physical obstruction by RNA or inadequate trypsin? To find an expected limit under the most favourable conditions RNAse has been used prior to overnight tryptic digest, trypsin increased to 0.5ug for 2e6 cells RBP equivalent, and digestion volume reduced to 50ul (from 100ul). We seek missed cleavage rates to be similar to that of the total inputs (100k cells whole proteome, 1ug trypsin) and for those total input samples to show a missed cleavage rate comparable to previous similar experiments (to confirm there are no batch issues with the new enzyme).
B. To explore the use of total inputs as a quantitative baseline for discriminating between RBPs more closely associated with non-adenylated RNA than adenylated RNA
See associated writeup for Expt.311 for a primer on how this is proprosed (ultimately, for this experiment, the underperformance of the silane groups precludes this analysis from being done properly).
Method:
See associated writeup for Expt.311
Custom Functions
jwrangle.importMixedFiles( )
I generally import everything I MIGHT use at the start and set up pathing using the OS-agnostic pathlib.
#### File utilities
import os
import pandas as pd
from pathlib import Path
from imp import reload
#### Data Wrangling
import copy
import numpy as np
#### RBP Suite Modules
import jwrangle
import jvis
import jinspect
import jtest
import jweb
#### Sequence Tools
from Bio import SeqIO
#### Graphical Packages
import upsetplot as upset
import seaborn as sns
import matplotlib.pyplot as plt
import altair as alt
#### define working directories
cwd = Path(os.getcwd())
base_path = Path(os.path.join(*cwd.parts[:cwd.parts.index('experiments')]))
#### MaxQuant proteinGroups & evidence files
MQ_folder = jwrangle.importMixedFiles(cwd / 'MaxQuant')
MQ_folder.keys()
pGroups = MQ_folder['proteinGroups.txt']
evidence = MQ_folder['evidence.txt']
#### Inspect MQ setup
MQ_folder['parameters.txt'].head(9)
Custom Functions
jwrangle.MQ_writeMetadata( )
Metadata tabulates the test conditions for ALL experiments that shared the same MQ search and thus all experiments that comprise the MQ outputs. Metadata can also be done in a spreadsheet program.
The metadata table gives users the opportunity to rename samples and define the experimental parameters for the data. This task can be expecially complex for MaxQuant because a unified output is generated even if distinctly separate experiments are searched as a batch and with different parameters applied. The function jwrangle.MQ_writeMetadata( ) will take a metadata table, rename all samples in the proteinGroups and evidence files, assign alternative filenames, and save new copies to be used in future analyses.
#### Inspect column names
colnames = list(pGroups.columns.values)
#### Derive experiment names as a list
experiment_names = []
for i in colnames:
if 'Intensity ' in i:
experiment_names.append(i.replace('Intensity ', ''))
#### Create a list of associated conditions
condition = ['TI_1ul_nCL']*6 + ['TI_5ul_254']*6 + ['OdT_nCL']*6 + ['OdT_254']*6 + ['Sil_nCL']*6 + ['Sil_254']*6
#### Create a list of associated replicate identifiers
replicate = ['A','B','C','D','E','F']*6
#### Create a more reader friendly list of sample names
samples = ['01_TI_1ul_nCL_A', '02_TI_1ul_nCL_B', '03_TI_1ul_nCL_C', '04_TI_1ul_nCL_D', '05_TI_1ul_nCL_E', '06_TI_1ul_nCL_F',
'07_TI_5ul_254_A', '08_TI_5ul_254_B', '09_TI_5ul_254_C', '10_TI_5ul_254_D', '11_TI_5ul_254_E', '12_TI_5ul_254_F',
'13_OdT_nCL_A', '14_OdT_nCL_B', '15_OdT_nCL_C', '16_OdT_nCL_D', '17_OdT_nCL_E', '18_OdT_nCL_F',
'19_OdT_254_A', '20_OdT_254_B', '21_OdT_254_C', '22_OdT_254_D', '23_OdT_254_E', '24_OdT_254_F',
'25_Sil_nCL_A', '26_Sil_nCL_B', '27_Sil_nCL_C', '28_Sil_nCL_D', '29_Sil_nCL_E', '30_Sil_nCL_F',
'31_Sil_254_A', '32_Sil_254_B', '33_Sil_254_C', '34_Sil_254_D', '35_Sil_254_E', '36_Sil_254_F']
#### Define the experiment group each sample belongs to.
MQ_groups = ['TI_lo']*6 + ['TI_hi']*6 + ['OdT']*12 + ['Sil']*12
#### Create metadata dataframe and inspect
expt_df = pd.DataFrame(
{'experiment': experiment_names,
'condition': condition,
'replicate': replicate,
'sample':samples,
'measure':['Intensity']*len(samples), # adding this column allows our metadata file to be compatible with Proteus
'MQgroups':MQ_groups
})
expt_df
# MQ_expt298 = jwrangle.MQ_writeMetadata(pGroups, evidence, expt_df, 'e311', cwd)
Functions
jweb.mapAnyID( )
jwrangle.importMixedFiles( )
MaxQuant does a good job of assigning a Gene name to each protein group. Presumably these gene names come from the FASTA. However:
To avoid these problems we will remap the Majority protein IDs to ENTREZ gene IDs. jweb.mapAnyID( ) will retrieve all possible genes for each protein group, and will also select a primary ID to singularly represent the group by a consistent method. This is a very flexible function, see help( ) for further explanation. From this point, the MQ 'Gene names' column will no longer be necessary. This function can also handle ID mapping to and from almost any convention.
Ensuring our proteins have a consistent gene naming strategy is essential for inter-experiment comparison and the later use of set methods. It also creates a standard that can be applied for accurately mapping RNA-Seq results and thus aid in future mapping of protein-RNA partners.
#### If not already loaded, read in the metadata-adjusted files
metadata = pd.read_csv(cwd / 'metadata' / 'e311_metadata.csv', index_col = 0)
pGroups = pd.read_csv(cwd / 'MaxQuant' / 'e311_proteinGroups_metalabeled.txt', delimiter = '\t')
evidence = pd.read_csv(cwd / 'MaxQuant' / 'e311_evidence_metalabeled.txt', delimiter = '\t')
#### Dynamically remap gene names in our proteinGroups file and save a copy
# pGroups_remap = jweb.mapAnyID_gPro(pGroups['Majority protein IDs'].tolist(), splitstr = [';', '-'], geneProductType = 'protein',
# gConvertOrganism = 'hsapiens', gConvertTarget = 'ENTREZGENE', writetopath = [cwd, 'pGroups_remap'], writeTargetsAsList = 'NO')
#### If not already loaded, read in the remapped proteinGroups file
pGroups_remap = jwrangle.importMixedFiles(cwd / 'downloads' / 'pGroups_remap', dropSuffix = 'yes')
pGroups_remap.keys()
#### jwrangle.importMixedFiles() returns a dictionary where keys = files. We want the 'id_map' table created by jweb.mapAnyID_gPro().
#### We'll rename the Query column and drop duplicates so the table can be merged with our proteinGroups table.
id_map = pGroups_remap['id_map'].rename(columns={'Query': 'Majority protein IDs'}).drop_duplicates()
id_map.head(2)
#### Now use merge to add these new columns to our proteinGroups table
pGroups_map = pd.merge(pGroups, id_map, on='Majority protein IDs', how='left')
#### Check the tables are merged by viewing column elements from each.
pGroups_map[id_map.columns.tolist() + ['Peptide IDs']].head(2)
Functions
jinspect.MQ_getContaminants( )
MQ_getContaminants_sbplot( )
jwrangle.importMixedFiles( )
We can extract the conaminants from our proteinGroups file using jinspect.MQ_getContaminants( ). These extracted table will return log2(iBAQ values).
Contaminants can then be reviewed with _MQ_getContaminantssbplot( ).
#### Extract contaminants
contaminants = jinspect.MQ_getContaminants(pGroups_map, metadata)
contaminants.head(2)
#### Visaully inspect contaminants
jvis.MQ_getContaminants_sbplot(contaminants, metadata, width = 1, length = 2, layout = 'single')
I expect this is a result of using RNAses T1 and A from a non-HPLC purified source.
The presence of streptavidin in the total input samples is puzzling.
RNAses were introduced to investigate the improvement of digestion efficiency. They are unlikely to be used routinely because they introduce background signal to the nCL groups.
Functions
jinspect.MQ_getMissedCleavages( )
jvis.CommonPalettesAsHex
jvis.BarPlotByGroup_sbplot( )
Assessing missed cleavages is an essential metric for understanding the quality of the tryptic digestion. This data is recorded in the evidence file.
_jinspect.MQgetMissedCleavages( ) will return a long form data table that can easily be used for plotting.
The dictionary jvis.CommonPalettesAsHex contains a number of palettes that are common to both matplotlib and ggplot (from R). These are provided to ensure consistency is easy to achieve across both languages.
We'll plot the missed cleavages with the generic function jvis.BarPlotByGroup_sbplot( )
#### Extract the missed cleavage data into a long form table for plotting
MissedCleavages = jinspect.MQ_getMissedCleavages(evidence, metadata, drop_contaminants = True)
MissedCleavages.sort_values(by=['sample'], inplace = True)
MissedCleavages
### Select a colour palette
cpal = jvis.CommonPalettesAsHex
set2_paired = []
for i in cpal['Set2_qual']:
set2_paired.append(i)
set2_paired.append(i)
#### Plot the grouped data points
sns.set_style('whitegrid')
jvis.BarPlotByGroup_sbplot(MissedCleavages, x_col = 'group', y_col = '% Missed Cleavages', title = '% Missed Cleavages', pal = set2_paired)
Good digestion efficiency might be the individual or combined result of
Moving forward, I think we can assume there is a great deal of protein
Next time increase to 1ug Trypsin per 10e6 cells in 50ul without RNAse pretreatment, maintain 0.5ug for RBP extgracted from 5e6 or fewer cells
Functions
jwrangle.MQ_getThreePassFilter( )
SeqIO.parse( )
After QC we no longer want the contaminants in our data. jwrangle.MQ_getThreePassFilter( ) will remove reverse peptides, contaminants, and only identified by site from MQ tables.
The filter will also accept customised exclusion lists in case users have added odd protein species to the search FASTA tables. In this particular experiment we added to the human FASTA, RNAse proteins and the large T antigen. The former as 1) a check that dynamic range is not being overwhelmed and 2) as an quantitative spike-in control to compare tryptic efficiency and the sample recovery across samples following C18 cleanup.
#### Map the location of the custom FASTA elements
os.listdir(base_path / 'my_resources' / 'FASTA')
#### Create a list of the non-human proteins that were added to the custom FASTA genome search.
new_cont = []
with open(base_path / 'my_resources' / 'FASTA' / "custom_proteome_elements.fasta", "r") as handle:
for record in SeqIO.parse(handle, "fasta"):
new_cont.append(record.id.split('|')[1])
#### Remove all unwanted contaminants and IDs from the proteinGroups table
pGroup_clean = jwrangle.MQ_getThreePassFilter(pGroups_map, custom_exclusion = new_cont)
#### Inspect the cleaned dataframe
pGroup_clean[['ENTREZGENE_gPro primary'] + [i for i in pGroup_clean.columns if 'iBAQ' in i]].head(2)
Functions
jinspect.MQ_dropDuplicateIDs( )
The next step focuses on improving confidence in the quality of our data. This is done by applying jinspect.MQ_dropDuplicateIDs( ) which has the below effects:
#### Drop duplicates and apply LFQ filter
filter_dict = jinspect.MQ_dropDuplicateIDs(pGroup_clean, metadata, prefix = 'Peptides', ID = 'ENTREZGENE_gPro primary', pool = 'measure', drop_ID = 'None',
keep_PoolCalcs = False, applyLFQ_filter = ['Intensity', 'iBAQ'])
#### Inspect filter dictionary
filter_dict.keys()
#### The df_keep value contains our targets, df_droprows conatins the discarded duplicates. Assign the df_keep value to a new variable and inspect.
pGroup_filtered = filter_dict['df_keep']
pGroup_filtered.head(2)
Functions
jtest.getDistanceMatrix( )
jvis.MQ_showDendrogramQC_mplplot( )
A distance matrix function jtest.getDistanceMatrix( ) is provided for users who wish to apply different algorithms or create different visualisations.
I like the 'ward' method for distance calculations and using a dengrogram to confirm that clustering matches expectations and so use a prerolled function jvis.MQ_showDendrogramQC_mplplot( )
#### Confirm that clustering matches expectations
jvis.MQ_showDendrogramQC_mplplot(pGroup_filtered, 'LFQ intensity', metadata, 'QC clustering: ', grid = 'YES', fsize = (8, 8))
Clustering reveals expected results
Functions
jwrangle.Log2_ByPrefix( )
jwrangle.MQ_poolMulti( )
jvis.ViolinCompare_sbplot( )
Here we review normalisation effects on each sample within the condition groups; these are most easily interpreted after log2 transformation. We will transform all measures of interest with _jwrangle.Log2ByPrefix( ) and then pool all the values of interest, by condition, with _jwrangle.MQpoolMulti( ). The function _jvis.ViolinComparesbplot( ) will let use compare Intensity distribution on a per sample basis.
Normalisation is applied to LFQ values by MaxQuant and is a feature of its handling of label-free data. I've not seen a detailed explanation of how it works though so it is a leap of faith that Cox and Mann have selected an appropriate method.
Normalisation must be applied separately to nCL and cCL groups. This is unusal though necessary to avoid outrageous results. See expt313 for evidence.
#### Log2 transform available intensity values.
pGroup_log2 = jwrangle.Log2_ByPrefix(pGroup_filtered, 'LFQ intensity')
pGroup_log2 = jwrangle.Log2_ByPrefix(pGroup_log2, 'iBAQ')
pGroup_log2 = jwrangle.Log2_ByPrefix(pGroup_log2, 'Intensity')
pGroup_log2.replace(0,np.nan, inplace=True)
#### Create a long form dataset for each desired grouping
pool_SampleIntensity = jwrangle.MQ_poolMulti(pGroup_log2, metadata, melt_list = ['Intensity', 'LFQ intensity'], group = 'condition')
pool_SampleIntensity.keys()
#### Inspect the Intensity shifts generated by the LFQ normalisation algorithm. In MaxQuant Raw Intensity and iBAQ values are
#### not subjected to the LFQ normalisation calculations so this is a good way to spot any gross violations
sns.set_style('whitegrid')
jvis.ViolinCompare_sbplot(pool_SampleIntensity['TI_1ul_nCL'], title = 'TI_1ul_nCL: Normalisation Effects', ylabel = 'Log2(Intensity)', palette = ['#ff6666', '#99ccff'])
jvis.ViolinCompare_sbplot(pool_SampleIntensity['TI_5ul_254'], title = 'TI_5ul_254: Normalisation Effects', ylabel = 'Log2(Intensity)', palette = ['#ff6666', '#99ccff'])
jvis.ViolinCompare_sbplot(pool_SampleIntensity['OdT_nCL'], title = 'OdT_nCL: Normalisation Effects', ylabel = 'Log2(Intensity)', palette = ['#ff6666', '#99ccff'])
jvis.ViolinCompare_sbplot(pool_SampleIntensity['OdT_254'], title = 'OdT_254: Normalisation Effects', ylabel = 'Log2(Intensity)', palette = cpal['Set3_qual'])
jvis.ViolinCompare_sbplot(pool_SampleIntensity['Sil_nCL'], title = 'Sil_nCL: Normalisation Effects', ylabel = 'Log2(Intensity)', palette = cpal['Set2_qual'])
jvis.ViolinCompare_sbplot(pool_SampleIntensity['Sil_254'], title = 'Sil_254: Normalisation Effects', ylabel = 'Log2(Intensity)', palette = cpal['Set2_qual'])
No weird sample variations (except perhaps sil-ncl B/F
No outrageous normalisation adjustments
Functions
jwrangle.MQ_poolDataByCondition( )
jvis.BoxPlotByColumn_sbplot( )
Next we will compare intensity and sequence coverage between groups. Log2 transformation has already been performed so we need only use jwrangle.MQ_poolDataByCondition( ) to create the appropriate long form dataset for plotting with jvis.BoxPlotByColumn_sbplot( ).
#### Pool data into a single long form dataset
pooled_dfDropGroupOne = jwrangle.MQ_poolDataByCondition(pGroup_log2, metadata, prefix_list = ['Intensity', 'Sequence coverage'])
#### Compare Intensity distribution using a box and whisker plot
sns.set_style('whitegrid')
jvis.BoxPlotByColumn_sbplot(pooled_dfDropGroupOne, 'Intensity: ', 'Intensity')
#### Compare Sequence coverage using a box and whisker plot
sns.set_style('whitegrid')
jvis.BoxPlotByColumn_sbplot(pooled_dfDropGroupOne, 'Sequence coverage: ', 'Sequence coverage %')
These results are consistent with expectations.
Previously I had theorised that the slightly lower sequence coverage typically found in RNP extracted samples could be due to excess RNA affecting tryptic efficiency- but the 1ul vs 5ul Total Input samples show that the sequence coverage is most likely being reduced only as a result of less material being present. A pattern that also holds true for nCL vs cCL comparisons.
Functions
jinspect.MQ_getSumBySample( )
jvis.BarPlotByGroup_sbplot( )
To sum the total peptides observed across all proteins use _jinspect.MQgetSumBySample( ). These sums will be returned as a modified metadata table.
Plotting these by group is easily done with jvis.BarPlotByGroup_sbplot( ). The plotting order is determined by the metadata ordering.
In this case we are inspecting the number of peptides detected after having removed contaminants- thus if some spike-in proteins were removed, i.e. in this case RNAse treatments, they will not contribute to the peptide count. To look at the replicability of these spike-ins, we would reach back to the 'df_droprows' table generated by jinspect.MQ_dropDuplicateIDs( ) in section 7.
#### Extract the total peptides observed per sample
metaStats = jinspect.MQ_getSumBySample(pGroup_log2, metadata, freqList = ['Peptides'], measure = False)
metaStats
#### Plot the sum peptides
sns.set_style('whitegrid')
jvis.BarPlotByGroup_sbplot(metaStats, x_col = 'condition', y_col = 'Peptides', title = 'Sum Peptides vs Silica Capture', pal = set2_paired,
errorbars = 'SEM')
Obviously the 5ul total input injections return far more peptides. But do they return more proteins?
Functions
jinspect.MQ_getFrequencyBySample( )
One gene can encode for many proteins that often share regions of similarity. As for illumina-based RNA-Seq, however, shotgun proteomics can rarely assign a peptide species to a singular protein. In MaxQuant these are called proteinGroups. Because we have do not require protein-specific results, and gene identity is more stable, our gene count describes the groups to which our detected proteins have been be assigned. Thus gene here is being detected by protein product, just as it would be detected by RNA product in RNA Seq; none of these 3 are synonymous. To be clear, this is a count and not a measure.
Gene frequency is defined by the summed observations per protein regardless of intensity value and this data is extracted to our modified metadata with jinspect.MQ_getFrequencyBySample( ) .
A typical MQ search will yield identical protein counts (though different values) for Intensity and iBAQ*. LFQ frequencies will vary depending on the search settings:
Notes
* Why protein counts should be identical I don't know. The original iBAQ paper stipulates rules for the inclusion of a protein in the iBAQ calculation but MaxQuant doesn't seem to apply them.
** Previously I tested LFQ min ratio at 1 peptide. At 1 minimum peptide there was unexpected QC clustering. Possible explanations for this are explained in section 7 and are cleaned up by jinspect.MQ_dropDuplicateIDs( ) function. We can expect this function to greatly reduce qualifying IDs (~20% fewer), especially in the QE samples, but I think the trade-off is worth it because we gain 1) a more robust ID check and 2) the same search can be used for LFQ based checks of dynamic changes, i.e. comparing more than one group of cCL captures for biological changes.
#### Count the number of unique
metaStats = jinspect.MQ_getFrequencyBySample(pGroup_log2, metaStats, freqList = ['Intensity', 'iBAQ', 'LFQ intensity'], measure = False)
metaStats
#### Plot the counts
sns.set_style('whitegrid')
jvis.BarPlotByGroup_sbplot(metaStats, x_col = 'condition', y_col = 'Intensity', title = '# Genes Detected By Group', pal = set2_paired, ylabel = 'Unique Genes',
errorbars = 'SEM')
This experiment tested some minor changes to the RBP purification protocol. Both OdT and Silane captures were affected by:
The silane capture also featured:
Both yielded comparatively high backgrounds on the nCL samples (c. 15-20%). In the next experiment (expt.314) we will:
Functions
jwrangle.MQ_getSliceByPrefix( )
jvis.showPearsonRegression_altair( )
The function _jwrangle.MQgetSliceByPrefix( ) provides a convenient means of extracting values of a specific group.
We can then use _jvis.showPearsonRegressionaltair( ) to perform pairwise comparisons between each member of those groups. This function is specifically applied to genes with shared intensities- genes exclusive to one sample or the other, represented by vertical or horizontal datapoints, are plotted but excluded from the pearson calculation.
#### Extract the intensity values as a dictionary where keys = groups
Intensity_Dict = jwrangle.MQ_getSliceByPrefix(pGroup_log2, metadata, 'Intensity', group = 'condition', add_col = None)
Intensity_Dict.keys()
#### Check replicate consistency across all within group pairs
jvis.showPearsonRegression_altair(Intensity_Dict['TI_5ul_254'], mark_color = set2_paired[2])
jvis.showPearsonRegression_altair(Intensity_Dict['OdT_nCL'], mark_color = set2_paired[4])
jvis.showPearsonRegression_altair(Intensity_Dict['OdT_254'], mark_color = set2_paired[4])
jvis.showPearsonRegression_altair(Intensity_Dict['Sil_nCL'], mark_color = set2_paired[6])
jvis.showPearsonRegression_altair(Intensity_Dict['Sil_254'], mark_color = set2_paired[6])